(19) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(12) 



(43) Date of publication: 

11.10.2000 Bulletin 2000/41 

(21) Application number: 00303009.5 

(22) Date of filing: 10.04.2000 



(n) EP 1 043 689 A2 

EUROPEAN PATENT APPLICATION 

(51) lntCl7: G06T 7/00 



(84) 


Designated Contracting States: 


(72) 


Inventors: 


AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


• 


Watanabe, Atsushi 




MC NL PT SE 




Suginami-ku, Tokyo 168-0072 (JP) 




Designated Extension States: 


• 


Arimatsu, Taro 




AL LT LV MK RO SI 




Minamitsuru-gun, Yamanashi 401-0310 (JP) 


(30) 


Priority: 08.04.1999 JP 10188599 


(74) 


Representative: Billington, Lawrence Emlyn 




Haseitine Lake & Co. 


(71) 


Applicant: FANUC LTD 




Imperial House 




Minamitsuru-gun, Yamanashi 401-0597 (JP) 




15-19 Kingsway 






London WC2B 6UD (GB) 



(54) Image processing apparatus 

(57) An image processing apparatus is capable of 
detecting position and posture of individual workpieces 
randomly arranged in a pile and having identical shapes. 
Reference models are created from two-dimensional 
images of a reference workpiece captured in a plurality 
of directions by a camera and stored. Also, the relative 
positions/postures of the workpiece with respect to the 
camera at the respective image capturing are stored. 
An image of a pile of workpieces is captured by the cam- 
era to obtain a two-dimensional image and the position/ 
posture of the camera at the image capturing is stored. 



An image of a workpiece matched with one reference 
model is selected by matching processing of the refer- 
ence model with the captured image. A three-dimen- 
sional position/posture of the workpiece with respect to 
the camera is obtained from the image of the selected 
workpiece, the selected reference model and position/ 
posture information associated with the reference mod- 
el. A picking-up operation for picking out a respective 
workpiece from a randomly arranged pile can be per- 
formed by a robot, based on the position/posture of the 
workpiece. 
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Description 



[0001] The present invention relates to an image 
processing apparatus tor detecting three-dimensional, 
position and posture (orientation) of an object, and in 
particular to an image processing apparatus suitable for 
use in a bin-picking operation for taking out a workpiece 
one by one from a pile of workpieces using an industrial 
machine such as a robot. 

[0002] The operation of taking out an individual work- 
place from a randomly arranged pile of workpieces or 
an aggregation of workpieces contained in a container 
of a predetermined size, which workpieces have identi- 
cal shapes and different three-dimensional positions/ 
postures, has been performed manually. In storing 
workpieces in a pallet or placing workpieces at a prede- 
termined position in a machine or a device using a (ded- 
icated) robot, since it has been impossible to directly 
take out individual workpieces one by one from the ran- 
domly arranged pile of workpieces by the dedicated ro- 
bot, it has been necessary to rearrange the workpieces 
in advance so as to be picked out by the robot. In this 
rearrangement operation, it has been necessary to take 
out an individual workpiece from the pile manually. 
[0003] The reason why individual workpieces having 
identical shapes and different three-dimensional posi- 
tions/postures cannot be picked out by a robot from a 
randomly arranged pile of workpieces or an aggregation 
of workpieces contained in a container is that the posi- 
tion/posture of individual workplaces in the pile or the 
aggregation cannot be recognised/so that a robot hand 
cannot be placed to a suitable position/posture at which 
the robot hand can hold the individual workpiece. 
[0004] An object of the present invention is to provide 
an image processing apparatus capable of detecting 
three-dimensional position and posture of individual ob- 
jects in a randomly arranged pile or an aggregation in a 
container of a predetermined region, which have iden- 
tical shapes and different three-dimensional positions/ 
postures. 

[0005] An image processing apparatus of the present 
invention comprises an image capturing device; and a 
memory storage reference model based on image data 
of a reference object captured by the image capturing 
device in a plurality of directions, and storing information 
of the capturing directions to be respectively associated 
with the reference models. The reference object may be 
an object for detection itself or an object having a shape 
identical to that of the object for detection. 
[0006] The image processing apparatus also com- 
prises a processor for performing matching processing 
of image data containing an image of the object for de- 
tection captured by the image capturing device with im- 
age data for reference models to select an image of an 
object matched with one of the reference models, and 
to obtain posture, or posture and position, of the object 
based on the selected image of the object, said one ref- 
erence model and the information of the direction asso- 



ciated with said one reference model. 
[0007] The reference models may be a part of the im- 
age data of the reference object or obtained by process- 
ing the image data of the reference object, 
s [0008] The image capturing device may be a camera 
for capturing two-dimensional image data, and in this 
case the image data of the reference object are captured 
by the image capturing device from a predetermined dis- 
tance. Alternatively, the image capturing device may be 
10 a visual sensor for capturing three-dimensional image 
data, and when the three-dimensional visual sensor is 
adopted the image data containing an image of the ob- 
ject of detection may be two-dimensional arrangement 
data including distance information from the object of 
is detection to the image capturing device, a part of said 
two dimensional arrangement data or a set of distance 

data. , 
[0009] The image capturing device may be attached 
to a wrist of a robot. Further, the image data of the ref- 
20 erence object can be captured in a place different from 
a place where the detection of the object is performed, 
and supplied to the image processing apparatus on line 
or off line. 

[0010] For a better understanding of the invention, 
25 and to show how the same may be carried into effect, 
reference will now be made, by way of example, to the 
accompanying drawings, in which:- 
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FIG. 1 is a diagram for showing a picking-up oper- 
ation by a robot to take out an individual workplace 
from a pile of workpieces using an image process- 
ing apparatus according to an embodiment of the 
present invention; 

FIGS. 2a-2d show examples of reference models; 
FIG . 3 is a block diagram of a principal part of a robot 
controller; 

FIG. 4 is a block diagram of the image processing 
apparatus according to an embodiment of the 
present invention; 

FIG. 5 is a flowchart of the processing for creating 
reference models; 

FIG. 6 is a flowchart of the processing for the pick- 
ing-up operation; 

FIG . 7 is a diagram showing an example of scanning 
motion of a visual sensor capable of obtaining dis- 
tance data; 

FIG. 8 is a diagram of the two-dimensional arrange- 
ment data containing distance data as image data 
obtained by the visual sensor; 
FIG. 9 is a flowchart of processing for obtaining the 
two-dimensional arrangement data. 1 



[0011] An embodiment in which an image processing 
apparatus of the present invention is used in combina- 
55 tion with a robot system will be described. In this em- 
bodiment, an image of a pile of workpieces, which are 
objects for detection having identical shapes and ran- 
domly arranged as shown in FIG. 1, is captured by an 
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image capturing device (camera or visual sensor) 20, 
which is attached to a wrist of a robot RB and position 
and posture (orientation) of the individual workplaces 
are detected based on the captured image. For this pur- 
pose, images of a reference object, which is one of work- 
places W, subjected to a picking-up operation or an ob- 
ject having a shape identical to that of the workpiece W 
are captured in different directions by the image captur- 
ing device and reference models are created from the 
image data obtained by the image capturing and stored 
in advance. Matching processing between the image 
data obtained by capturing the image of the pile of work- 
places and the reference models is executed to select 
an image of one workpiece matched with one of refer- 
ence models, and a position/posture of the selected 
workpiece is determined based on the selected image 
of the workpiece in the image field of view the selected 
one of taught modes and the position/posture informa- 
tion being associated with the selected one of the refer- 
ence models. 

[0012] FIG. 3 is a block diagram showing a principal 
part of a robot controller 10 for use in the embodiment 
of the present invention. A main processor 1 , a memory 
2 including a RAM, a ROM and a nonvolatile memory 
(such as an EEPROM), an interface 3 for a teaching op- 
erating panel, an interface 6 for external devices, an in- 
terface 7 for an image processing apparatus and a servo 
control section 5 are connected to a bus 8. A teaching 
operating panel 4 is connected to the interface 3 for a 
teaching operating panel. , 

[001 3] A system program for supporting basic func- 
tions of the robot RB and robot controller 10 are stored 
in the ROM of the memory 2. Robot operation programs 
and their related determined data which are taught in 
accordance with various operations are stored in the 
nonvolatile memory of the memory 2. The RAM of the 
memory 2 is used for temporary storage of data for var- 
ious arithmetic operations performed by the processor 
1. 

[0014] The servo control section 5 comprises servo 
controllers 5al to 5an (n:sum of the number of all the 
axes of the robot including additional movable axes of 
a tool attached to a wrist of the robot), each composed 
of a processor, a ROM, a RAM, etc. Each servo control- 
ler performs position/velocity loop control and also cur- 
rent loop control for its associated servomotor for driving 
the axis, to function as a so-called digital servo controller 
for performing loop control of position, velocity and cur- 
rent by software. Each servomotor M1-Mn for driving 
each axis has its drive controlled according to outputs 
of the associated servo controller 5al-5an through the 
associated servo amplifier 5bl-5bn. Though not shown 
in FIG. 3, a position/velocity detector is attached to each 
servomotor Ml-Mn, and the position and velocity of each 
servomotor detected by the associated position/velocity 
detector is fed back to the associated servo controller 
5al-5an. Connected to the input/output interface 6 are 
sensors of the robot, and actuators and sensors of pe- 



ripheral devices. 

[0015] FIG 4 is a block diagram of the image process- 
ing apparatus 30 connected to an interface 7 of the robot 
controller 10. The image processing apparatus 30 com- 
5 prises a processor 31 to which a ROM 32 for storing a 
system program to be executed by the processor 31 , an 
image processor 33, an image-capturing- device inter- 
face 34 connected to the image capturing device 20, a 
MDI 35 with a display such as a CRT or a liquid crystal 
io display for inputting and outputting various commands 
and data, a frame memory 36, a nonvolatile memory 37, 
a RAM 38 for temporary storage of data and a commu- 
nication interface 39 for the robot controller are connect- 
ed. An image captured by the camera 20 is stored in the 
is frame memory 36. The image processor 33 performs 
image processing from images stored in the frame 
memory 36 on demand of the processor 31 so as to rec- 
ognise an object. The architecture and function of the 
image processing apparatus 30 itself is no way different 
20 form the conventional image processing apparatus. The 
image processing apparatus 30 of the present invention 
is different form the conventional one in that reference 
models as described later are stored in the nonvolatile 
memory 37 and pattern matching processing is per- 
25 formed on an image of a pile of workpieces W captured 
by the image capturing device 20 using the reference 
models to obtain the position and posture of a workpiece 
W. 

[001 6] The image capturing device 20 is used for ob- 

30 taining image data, as described later, and may be a 
CCD camera for obtaining two-dimensional images data 
or a visual sensor capable of obtaining three-dimension- 
al image data including distance data. In the case of us- 
ing the CCD camera, the image data is obtained by a 

35 conventional method based on two-dimensional images 
captured by the CCD camera, but in the case of the vis- 
ual sensor capable of obtaining three-dimensional data 
including distance data, two-dimensional arrangement 
data with distance data between the sensor and an ob- 

40 ject is obtained. A visual sensor for obtaining the three- 
dimensional data including distance data is known, for 
example, from three-dimensional visual sensors of a 
spot light scanning type disclosed in Japanese Patent 
Publication No. 7-270137, and the summary of the 

45 three-dimensional visual sensor is described below. 
[0017] This visual sensor detects a three-dimensional 
position of an object by irradiating a light beam to form 
a light spot on the object for scanning the object in two 
different directions (X direction and Y direction) and by 
50 detecting the light reflected on the object by a position 
sensitive detector (PSD). Three dimensional position of 
the object is measured by a calculation using the re- 
spective inclination angles Gx,0y of mirrors for scanning 
and an incident positions of the reflected light beam on 
55 the PSD. 

[0018] Referring to FIGS. 7-9, a method of obtaining 
two-dimensional arrangement data including distance 
data using the three-dimensional visual sensor will be 
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explained briefly. 

[0019] Scanning range (measuring range) on an ob- 
ject is set in advance, and an inclination angle 9x, ©y of 
the mirrors is controlled discretely. As shown in FIG. 7, 
the scanning is performed from a point (1,1) to a point 

( 1 , n), from a point (2, 1 ) to a point (2, n) from a point 

(m, 1 ) to a point (m, n) on the X-Y plane within the scan- 
ning range, to measure three-dimensional positions of 
each reflected point on the object Also, a distance Z (i, 
j) between the sensor and the reflection point (i, j) on 
the object is obtained and stored in the RAM38 of the 
image processing apparatus 30. Thus, the image data 
is obtained as two-dimensional arrangement data in- 
cluding the distance data Z (i, j) between the sensor and 
the reflection point on the object as shown in FIG. 8. 
[0020] FIG. 9 is a flowchart of processing to be exe- 
cuted by the processor 31 of the image processing ap- 
paratus 30 for obtaining the image data. 
[0021] First, indexes i and j are respectively set to "1 " 
(Step 300) and the inclination angle (0x, 6y) of the mir- 
rors is set to (x1 , yl) to direct to the start point (1,1) and 
an irradiation command with the inclination angle is 
send to the sensor 20 (Steps 301-303). The sensor ir- 
radiates a light beam with the mirrors set at the inclina- 
tion angle. The signal representing the image captured 
by the PSD is sent to the image processing apparatus 
30. The processor 31 of the image processing appara- 
tus 30 calculates the position of the reflection point on 
the object of the signal from the PSD and the inclination 
angle (Ox, Ay) of the mirrors to obtain the distance Z (i, 
j) between the sensor and the position of the reflection 
point on the object. This value Z (L j) is stored in the 
RAM 38 as the two-dimensional arrangement data [i, j] 
(Step 304, 305). The calculation for obtaining the posi- 
tion of the reflection point and the distance Z (i, j) may 
be performed by the sensor 20. 

[0022] Then, the index i is incrementally increased by 
"V and the inclination angle ex of the mirror for X-axis 
direction scanning is increased by the predetermined 
amount Ax (Step 306, 307). It is determined whether or 
not the index i exceeds the set value n (Step 308). If the 
index i does not exceed the set value n, the procedure 
returns to Step 303 and the processing from Step 303 
to Step 308 is executed to obtain the distance Z (i, j) of 
the next point. Subsequently, the processing of Steps 
303-308 are repeatedly executed until the index i ex- 
ceeds the set value n to obtain and store the distance Z 
(i, j) of the respective points (1 , 1 ) to (1 , n) shown in Fl G. 
7. 

[0023] If it is determined that the index i exceeds the 
set value n in Step 308 : the index i is set to "1 " and the 
index j is incrementally increased by n 1° to increase the 
inclination angle 0y of the mirror for Y-axis direction 
scanning (Steps 309-31 1 ). Then, it is determined wheth- 
er or not the index j exceeds the set value m (Step 31 2) 
and if the index j does not exceed the set value m, the 
procedure returns to Step 302 to repeatedly executes 
the processing of Step 302 and the subsequent Steps. 



[0024] Thus, the processing from Step 302 to Step 
31 2 is repeatedly executed until the index j exceeds the 
set value m. If the index j exceeds the set value m, the 
points in the measurement range (scanning range) 

s shown in FIG. 7 have been measured entirely, the dis- 
tance data Z (1 , 1 ) - Z (m, n) as two dimensional arrange- 
ment data are stored in the RAM28 and the image data 
obtaining processing is terminated. A part of the image 
data of two dimensional arrangements or a plurality of 

10 distance data can be obtained by appropriately omitting 
the measurement of the distance for the index i. 
[0025] The foregoing is a description on the process- 
ing for obtaining two dimensional arrangement data as 
image data using the visual sensor capable of measur- 

is jng the distance. Using the two-dimensional arrange- 
ment data obtained in this way as image data, creation 
of reference models and detection of position and pos- 
ture (orientation) of an object can be performed. In order 
to simplify the explanation, the following description will 

20 be made assuming that a CCD camera 20 is used as an 
image capturing device and the two dimensional image 
data obtained by capturing image of the object by this 
camera 20 is used. 

[0026] Processing for creating reference models will 
25 be explained referring to FIGS. 2a-2d and FIG. 5. FIG. 
5 is a flowchart showing processing for teaching refer- 
ence models to the image processing apparatus 30 ac- 
cording to the present invention. 

[0027] One reference workpiece (one o\ the workpiec- 
30 es W as object for robot operation or a workpiece having 
a three-dimensional shape identical to that of the work- 
piece W) is prepared for creating reference models. A 
first (0-th) position/posture of the reference workpiece 
at which the camera 20 attached to a distal end of a robot 
35 wrist captures the image of the object is set, and an axis 
of rotation and rotation angles with respect to the first 
(0-th) position/posture are set in order to determine the 
subsequent positions/postures of the reference work- 
piece. In addition, the number of positions/postures of 
40 the workpiece at which the camera 20 captures the im- 
age of the object is set. In this example, information of 
both position and posture is used. However it is suffi- 
cient for creating reference models to use only posture 
(orientation) information if the demanded precision of 
45 position is not high. 

[0028] As shown in FIGS. 2a to 2d, in this example, 
images of the reference workpiece are captured from 
four different directions and reference. models are cre- 
ated based on the four image data. As shown in FIG. 
50 2a, an image of the reference workpiece is captured 
from the direction of a Z-axis of a world coordinate sys- 
tem at 0-th position/posture to create 0-th reference 
model. For setting the subsequent positions/postures, 
an axis perpendicular to an optical axis of the camera 
55 and passing a central point of the workpiece (origin of a 
work coordinate system set to the workpiece) and rota- 
tion angles of the workpiece along the rotation axis are 
set for this camera position. Since the optical axis of the 
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camera is set parallel to the 2 axis of the world coordi- 
nate system, an axis parallel to either the X-axis or the 
Y-axis of the world coordinate system, which is perpen- 
dicular to the Z axis, can be selected and the workpiece 
is rotated around the rotation axis at the workpiece po- 
sition. 

[0029] In the example, an axis parallel to the X-axis 
of the world coordinate system is set as the rotation axis, 
and for the position/posture shown in FIG. 2b, the rota- 
tion angle of 30° is set to rotate the workpiece by 30° 
with respect to the camera along the rotation axis. A first 
reference model is created based on the image data of 
the workpiece at the position/posture shown in FIG. 2b. 
Similarly, as shown in FIGS. 2c and 2d, the workpiece 
is rotated by 60° and90°, respectively, along the rotation 
axis for capturing images of the workpiece to create 2nd 
and 3rd reference models. 

[0030] In this example, rotation angles of zero degree, 
30 degrees, 60 degrees and 90 degrees are set for cre- 
ating four reference models. The dividing range of the 
rotation angles may be set more finely and/or range of 
the rotation angle may be set greater to create more ref- 
erence models for more precise detection of the posi- 
tion/posture of the workpiece. 

[0031] The processing for creating the four reference 
models will be explained referring to flowchart of FIG. 5. 
[0032] As described above, the 0-th position/posture 
of the robot at which the camera 20 captures the image 
of the object, and the rotation axis and the rotation an- 
gles with respect to the 0-th position/posture are set in 
advance in order to determine the subsequent positions/ 
postures of the reference workpiece, and also the 
number of the subsequent positions/postures of the 
workpiece are set. For intelligible explanation, it is as- 
sumed that an optical axis of the camera is parallel to 
the Y-axis of the world coordinate system and that a po- 
sition where the X-axis and Y-axis coordinate values are 
identical to those of the reference workpiece and only 
the Z-axis coordinate value is different from that of the 
position of the reference workpiece is taught to the robot 
as the 0-th image capturing position for obtaining the 
0-th reference model. Further, the positions of the robot 
where the camera is rotated with respect to the refer- 
ence workpiece by 30 degrees, 60 degrees and 90 de- 
grees along the axis passing the central point of the ref- 
erence workpiece and parallel to the X-axis of the world 
coordinate system are set as the 1 st, 2nd and 3rd image 
capturing position, and the number N of the image cap- 
turing positions is set to "4." 

[0033] When a command for creating reference mod- 
els is inputted from teaching operation panel 4, the proc- 
essor 1 of the robot controller 10 sets a counter M for 
counting the number of the image capturing to B 0 n (Step 
100). The robot is operated to have the M-th position/ 
posture and a command for image capturing is output- 
ted to the image processing apparatus 30 (Step 101). 
In response to this command, the image processing ap- 
paratus 30 performs capturing of an image of the refer- 



ence workpiece with the camera 20 and the captured 
image data is stored in the frame memory 36. Further, 
relative position/posture of the workpiece with respect 
to the camera is obtained and stored in the nonvolatile 
5 memory 37 as relative position/posture of M-th refer- 
ence model, and a data-captured signal is sent to a robot 
controller (Step 103). Thus, position/posture of the 
workpiece in a camera coordinate system set to the 
camera is obtained from the position/posture of the cam- 
to era and the position/posture of the reference workpiece 
in the world coordinate system when capturing the im- 
age by the camera, and is stored as the relative position/ 
posture of the workpiece with respect to the camera. For 
example, the position/posture of the workpiece in the 
is camera coordinate system is stored as [xO, yO, zO, <x0, 
p0, t0]c, where a, p and y mean rotation angle around 
X-, Y-, Z- axes, and "c" means the camera coordinate 
system. 

[0034] Upon receipt of the data-captured signal, the 
20 processor 1 of the robot controller 1 0 incrementally in- 
creases the value of the counter M by " 1 "(Step 104) 
and determines whether or not the value of the counter 
M is less than a set value N (=4) (Step 1 05). If the value 
of the counter M is less than the set value N. the proce- 
ss dure returns to Step 101 to move the robot to the M-th 
image-capturing position/posture. Thus, in the example 
as shown in FIGS. 2a-2d, the camera is successively 
turned by 30 degrees around the axis parallel to X axis 
of the world coordinate system and passing the work- 
30 piece position, and successively captures the image of 
the workpiece, and reference models and relativity po- 
sitions/postures of the camera with respect to the work- 
piece at the image capturing are stored. 
[0035] Processing of Steps 101-105 is repeatedly ex- 
35 ecuted until the value of the counter M equals to the set 
value N (=4), and the reference models and the relative 
positions/postures of the camera and the workpiece are 
stored in the nonvolatile memory 37. Thus, the refer- 
ence models created from the image data of the work- 
40 piece at the positions/postures shown in FIGS. 2a-2d 
are stored, and the relative positions/postures between 
the camera and the workpiece for respective reference 
models are stored as positions/postures of the work- 
piece W in the camera coordinate system as [x 0, y 0, z 
45 o, <xO, pO, yO]c,[x 1, y 1, z i,ccl, pi, y1]c, [x2, y 2, z 2, 
a2, p2, y2]c, and [x 3, y 3, z 3, a3, p3, y3]c. 
[0036] The reference models and the relative posi- 
tion/posture of the workpiece W and the camera 20 are 
stored in the nonvolatile memory 37 of the image 
so processing apparatus 30. In the above described em- 
bodiment, the reference models are created using a ro- 
bot, however, the reference models may be created by 
a manual operation without using a robot. In this case, 
the reference workpiece is arranged within the field of 
55 view of the camera connected to the image processing 
apparatus 30, and the images of the workpiece with dif- 
ferent postures are captured by the camera. The refer- 
ence models are created based on the image data and 



5 



9 



EP 1 043 689 A2 



10 



the relative positions/postures of the camera and the 
workpiece at the image capturing manually inputted, 
and are stored with the respective relative positions/ 
postures. 

[0037] The reference models may be created from a 
part of the image data of the reference object, and may 
be created by processing the image data of the refer- 
ence object. 

[0038] In addition, the reference models may be cre- 
ated based on the stored image data of the reference 
workpiece when detecting the position/posture of the 
objective workpiece, without creating and storing the 
reference models in advance. 

[0039] Hereinafter, a picking-up operation for taking 
out an individual workpiece by a robot from a pile of 
workpieces each having a shape identical to that of the 
reference workpiece will be described, as an example 
of a method of detecting three-dimensional position/ 
posture of an object, using the image processing appa- 
ratus 30 storing the reference models. 
[0040] FIG. 6 is a flowchart of the carrying out ol the 
picking-up operation. When a picking-up command is 
inputted into the robot controller 10 from the teaching 
operation panel 4, the processor 1 operates the robot 
RB to move the camera attached to the robot wrist to an 
image capturing position where a pile of workpieces is 
within a field of view of the camera 20 (Step 200). The 
three dimensional position/posture of the camera 20 on 
the world coordinate system at this image capturing po- 
sition is outputted to the image processing apparatus 
30, and an image capturing command is outputted (Step 
201 ). Upon receipt of the image capturing command, the 
processor 31 of the image processing apparatus 30 cap- 
tures an image of the pile of the workpieces W. to obtain 
image data of some workpieces W and store the data in 
the frame memory 36 (Step 202). 
[0041] Then, pattern matching processing is per- 
formed for the image data stored in the frame memory 
36 using one of the reference models (the first reference 
model) stored in the nonvolatile memory 37 so as to de- 
tect a workpiece W (Step 203). In this pattern matching 
processing, matching of the image data of the reference 
model with the image data of workpieces is performed 
on the basis of position, turn and scale. It is determined 
whether or not an object has a matching value equal or 
greater than the set value (Step 204). If an object having 
a matching value equal or greater than the set value is 
not detected, the procedure proceeds to Step 205 to de- 
termine whether or not the pattern matching is per- 
formed using all the reference models (1st to 4th refer- 
ence models). If the pattern matching using all the ref- 
erence models is not yet performed, further pattern 
matching is performed using another reference model 
(Step 206). 

[0042] If it is determined in Step 204 that an object 
having a matching value equal or greater than the set 
value with respect to any of the reference models is de- 
tected, the procedure proceeds to Step 207 to perform 



matching processing on the two-dimensional data of the 
detected workpieces W using every taught mode. In 
Step 208, the reference model having the largest match- 
ing value in the pattern matching processing is selected, 

5 andthe relative position/posture of theworkpiece Wwith 
respect to the camera 20 is determined based on the 
relative position/posture of the camera and the refer- 
ence workpiece stored for the selected reference model, 
and position, rotation angle and scale of the image of 

10 the workpiece in the matching processing, (Step 208). 
The position and posture (orientation) of the detected 
workpiece on the world coordinate system is determined 
from the position and posture of the camera 20 in the 
world coordinate system, which has been sent in Step 

is 201 , and the relative position/posture of the workpiece 
W with respect to the camera 20, and is outputted (Step 
209). Thus, since the relative position/posture of the 
workpiece W with respect to the camera 20 is the posi- 
tion/posture of the workpiece W in the camera coordi- 

20 nate system, the position and posture (orientation) of the 
detected workpiece W in the world coordinate system is 
obtained by an arithmetic operation of coordinate trans- 
formation using the data of the position/posture of the 
workpiece W in the camera coordinate system and the 

25 position/posture of the camera 20 in the world coordi- 
nate system (Step 209). 

[0043] The reference model having the highest 
matching value is selected in this embodiment, although 
a reference model of the rotation angle of zero degree 

30 (the O-th reference model) may be selected in accord- 
ance with precedents, or an object having the highest 
expansion rate of scale (the object which is nearest to 
the camera, i.e. located at the suit of the pile in this ex- 
ample) may be selected in accordance with precedents. 

35 [0044] The robot controller 10 operates the robot to 
perform a picking-up operation to grip and hold the de- 
tected workpiece W and move the held workpiece W to 
a predetermined position, based on the three-dimen- 
sional position/posture of the workpiece W (Step 210). 

40 Then, the procedure returns to Step 202 to repeatedly 
execute the processing of Step 202 and subsequent 
Steps. 

[0045] When all the workpieces have been picked-up 
from the pile of the workpieces, a matching value equal 
45 to or greater than the set reference value cannot be ob- 
tained in the pattern matching processing for all refer- 
ence models in Steps 203-206, and the picking-up op- 
eration is terminated. 

[0046] In the case where a pile of the workpieces can- 
50 not fall within the field of view of the camera 20, or in the 
case where it is not necessary to capture an image of a 
workpiece behind other workpieces by changing the ori- 
entation of the camera, the procedure may return to 
Step 200 when "Yes" is determined in Step 205, to move 
55 the camera to another position/posture at which an im- 
age of the object workpiece can be captured. 
[0047] In addition, in the case where the robot and the 
image processing apparatus 30 are used in combination 
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as in the foregoing embodiment, the robot controller 10 
may store the three-dimensional position/posture of the 
camera without outputting it to th e image processing ap- 
paratus 30 in Step 201 and the relative position/posture 
of the workpiece and the camera may be outputted from 
the image processing apparatus 30 to the robot control- 
ler 1 0 in Step 208 to execute the processing of Step 209 
in the robot controller 10. 

[0048] Further, in the case where a wide-angle lens is 
installed in the CCD camera as the image capturing de- 
vice, for example, there is possibility of judging the in- 
clination angle to be 30 degrees by influence of parallax 
when a workpiece of zero degree inclination is at a cor- 
ner of a field of view of the camera. In such a case, the 
camera may be moved parallel ly in accordance with the 
position of the workpiece in the field of view of the cam- 
era to a position right above the workpiece so that the 
effect of parallax is lost, and at this position the image 
capturing processing of Step 201 and the subsequent 
Steps in FIG. 6 is performed so that a false judgment is 
prevented. 

[0049] Furthermore, in order to obtain three-dimen- 
sional position/posture of an object workpiece whose 
three-dimensional position/posture is unknown without 
using a robot, the camera is arranged to capture an im- 
age of a pile of workpieces or a region containing the 
objective workpiece within a field of view of the camera, 
and the position/posture of the camera in the world co- 
ordinate system is inputted to the image processing ap- 
paratus 30 and an object detection command is issued 
to the image processing apparatus 30, to make the im- 
age processing apparatus 30 execute Steps 202-209 of 
FIG. 6. 

[0050] The image data for creating the reference 
models may be obtained at a place different form the 
place where the robot is installed. In this case, the image 
data may be supplied to the image processing appara- 
tus on line through a communication interface provided 
in the image processing apparatus, or may be supplied 
offline through a disc driver for reading a floppy disk, etc. 
[0051] According to the present invention, the posi- 
tion/posture of an object workpiece in a randomly ar- 
ranged pile of workpieces or an aggregation of work- 
pieces gathered in a predetermined region which have 
identical shapes and different three-dimensional posi- 
tions/postures is detected, to thereby enable a robot to 
automatically pick out an individual workpiece from such 
a pile or an aggregation. 



Claims 

1. An image processing apparatus for detecting pos- 
ture, or posture and position, of an object compris- 
ing: 

an image capturing device; 

a memory storage reference model created 



based on image data of a reference object cap- 
tured by said image capturing device in a plu- 
rality of directions, and storing information of 
the capturing directions to be respectively as- 

5 sociated with said reference models, said ref- 

erence object being the object for detection or 
an object having a shape identical to that of the 
object for detection; and 
a processor to perform matching processing of 

io image data containing an image of the object 

for detection captured by said image capturing 
device with image data for said reference mod- 
els to select an image of an object matched with 
one of said reference models, and to obtain 

is posture, or posture and position, of the object 

based on the selected image of the object, said 
one reference model and the information of the 
direction associated with said one reference 
model. 

20 

2. An image processing-apparatus according to claim 
1 , wherein said reference models comprises a part 
of the image data of the reference object. 

25 3. An image processing apparatus according to claim 
1 or 2, wherein said reference models are obtained 
by processing the image data of the reference ob- 
ject. 

30 4. An image processing apparatus according to any 
preceding claim, wherein said image capturing de- 
vice comprises a camera for capturing two-dimen- 
sional image data. 

35 5. An image processing apparatus according to claim 
4, wherein said image data of the reference object 
are captured by said image capturing device from 
a predetermined distance. 

40 6. An image processing apparatus according to any 
one of claims 1 to 3, wherein said image capturing 
device comprises a visual sensor for capturing 
three-dimensional image data. 

45 7. An image processing apparatus according to claim 
6, wherein said image data containing an image of 
the object for detection captured by said visual sen- 
sor are two-dimensional arrangement data includ- 
ing distance information from the object of detection 

so to the image capturing device, a part of said two- 
dimensional arrangement data or a set of distance 
data. 

8. An image processing apparatus according to an 
55 one of claims 1 through 7, wherein said image cap- 
turing device is attached to a robot. 

9. An image processing apparatus according to claim 
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1, wherein said image data of the reference object 
are captured in a place different from a place where 
the detection of the object is performed, and sup- 
plied to the image processing apparatus on line or 
off line. 
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